An Overview of NTCIR-5 QAC3

This paper provides an overview of NTCIR-5 QAC3 (Question Answering Challenge 3). QAC is a series of challenges for evaluating question answering technologies in Japanese. QAC3 follows the same course as QAC based on the success of the previous two workshops, with its task limited to that corresponding to QAC2 Subtask 3 aiming at the convergence of research resources for novel subjects. This task assumes interactive use of QA systems and evaluates, among other things, the abilities needed under such circumstances, i.e. proper interpretation of questions under a given dialogue context; in other words, context processing abilities such as anaphora resolution and ellipses handling (hereafter we refer to the task as the IAD task, where IAD stands for Information Access Dialogue, and to the whole workshop as QAC3). The IAD task in QAC3 is based on QAC2 Subtask 3 with several improvements, including elaboration of the scope of questions and answers and introduction of multi-grade evaluation and the concept of a correct answer set. In addition, a new WoZ method was devised and applied in the QAC3 test set construction. QAC3 had as many participants as QAC2 Subtask 3, and new trials and advances in existing methods were observed from the